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ABSTRACT 

The evaluation process provides valuable information 
so that practitioners may share and benchmark their efforts to 
improve and enhance the quality of the educational reform initiative 
called tech prep. Evaluation should be part of every phase of a tech 
prep initiative, from needs, assessment to completion. A tech prep 
program is typically at one of three evaluation stages: beginning 
programs, intermediate programs, and advanced programs. The 
evaluation design should be simple, understandable, and usable; 
dynamic; proactive; cost effective; quantifiable; transferable; and 
formative and summative in nature. Program focus components are areas 
of an educational program that are to be evaluated. Examples of 
program components that may be beneficial to consider when evaluating 
a. tech prep effort are as follows: administration and organization; 
articulation and collaboration; student program planning and 
implementation; curriculum development; marketing efforts; program 
effects; staff development; and evaluation strategies* A national 
evaluation of tech prep education programs funded under the Perkins 
Act is being conducted to identify effective practices. (YLB) 
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Tech Prep Decision-Making: Evaluation and Improvement Strategies 



Overview 

Prior to the widespread implementation of any program of instruction, such as Tech Prep, major 
questions concerning the program's vaiue need to be answered. Program evaluation is the vehicle by which 
decision-making information, addressing educational concerns is gathered and analyzed. 

The reauthorized Carl Perkins legislation mandates the development of "outcome indicators" (e.g., 
labor market, learning, and access indicators) designed to measure the performance of Tech Prep efforts 
within an education system. In addition, five major focus components have been identified as being 
important to the success of a vocational program in meeting proposed goals: student retention, job 
placement, competency gains in academics, work or job skill attainment, and vocational skill attainment. 
These focus components form the basis of the development guide and program component indicators to 
evaluate each of the components. 



Why Evaluate 

The approach to evaluation is based on the belief that planning and evaluation are two integrated 
aspects of the same process and evaluation is intended to assist the decision making process. Tech Prep 
evaluation is a process that focuses on the outcomes produced as a result of the Tech Prep efforts. It is 
also a process whicn provides for accountability. 

To conduct an evaluation, it is important to understand its meaning. Evaluation is the process of: 
(1) determining the decisions to be made; (2) selecting, collecting, and analyzing information needed in 
making these decisions; and (3) reporting the information to appropriate decision makers. Accountability 
is being responsible for accomplishing what one states they will accomplish relative to the resources 
available. 

The purpose of evaluation, then, is twofold. Evaluation data is collected to meet the concerns of 
practitioners and other stakeholders, and to provide useful information for making decision on program 
enhancements. Stakeholders include: counselors, business and industry, students, administrators, parents, 
etc. In addition, data is collected to meet the concerns of management audiences and to fulfill the legislative 
requirements of the states accountability system. 

The evaluation process provides valuable information so that we may share and benchmark our 
efforts to improve and enhance the quality of this important educational reform initiative called Tech Prep. 



When Should Evaluation Occur? 

Planning and evaluation are two integrated aspects of the same process. Evaluation should be a 
part of every phase of a Tech Prep initiative, from needs assessment to completion. For evaluation to attain 
its greatest value, we must be clear in terms of what we are doing and why we are doing it. Evaluation 
should not be some isolated activity that occurs after the fact A Tech Prep program is typically at one 
of three evaluation stages. These stages Include: 
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Stage One: Beginning Programs (1 - 2 years) 



In the beginning of Tech Prep programs, the focus of many program Improvement efforts Is on 
understanding the process of articulation and reporting the number of articulated classes/agreements. All 
postsecondary institutions have maintained a record of the number of articulated courses they offer. The 
record includes the number of signed articulation agreements and the courses and or program areas that 
have been articulate. During the first two years of a Tech Prep program evaluation, efforts are often 
focused on tho number of articulation agreements in place, and the number of articulated courses. Other 
outcome date, is often difficult to track during the initial two years of operation. 

Stage Two: Intermediate Programs (3 - 5 years) 

At the intermediate stage of program development, most administrators broaden the collection of 
discrete outcome indicators and systematically collect this information in anticipation of comprehensive 
program evaluations and potential program improvements. The information collected includes the change 
in the dropout rate and the number of: articulated programs, articulated courses, students enrolled in 
articulated classes, graduates and those who continue on to postsecondary institutions, and articulation 
credits earned. This stage begins to provide a basis for determining program "success" 

Stage Three: Advanced Programs (5 years +) 

At the advanced stage of program development, most administrators have identified a specific 
number of outcome indicators, established a formal process of data collection, and routinely published the 
results in an annual report. In some cases, computerized tracking systems have been developed, and 
eventually identify the students who have declared their Tech Prep major. The data base may include 
information such as the total number of courses in each high school, the articulated courses, credits 
available, and equivalent courses. 

Other data bases can be developed to measure the percent of enrollment by program of study, and 
percent of high school graduates attending postsecondary institutions, ideally the success of Tech Prep 
programs Is determined by conducting longitudinal evaluations and collecting information on multiple 
outcome indicators. Unfortunately most of the current evaluation efforts rely on limited number of outcome 
indicators such as enrollment figures and the number of articulated courses. The results of this type of data 
collection is that the items measure quantity, rather than quality (Dornsife, 1992). 



Dutton, Hammons, and Ownens (1993) identified seven basic principles/criteria which can be used 
in developing/or assessing any Tech Prep program. The evaluation design should be: 



Criteria for Evaluation Designs 



1. 
2. 
3. 
4. 
5. 
6. 
7. 



Simple, understandable, and useable, 

Dynamic, 

Pro-active, 

Cost effective, 

Quantifiable; 

Transferable, and 

Formative and summative in nature. 
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Simple, understandable and usable 

due to the critical need to involve many players In the Tech-Prep process (I.e. leaders of 
business, industry, and government; parents; students; policy-makers; school personnel- 
administrators and teachers, both academic and vocational; and, counselors) the evaluation 
plan must not use technology talk and educational jargon. 

everyone involved in the evaluation process needs to understand what the Tech Prep 
initiative is trying to accomplish. 

Dynamic 

evaluation plan must include the flexibility to grow as the Tech-Prep consortium grows. 

dynamic enough to examine additional areas of evaluation and allow for input and feedback 
from ali stakeholders. 

Pro-active 

identify expected outcomes, goals, and standards, 
monitor and report the status of these expectations. 

provide a means with which to identify and implement improvement strategies. 
Cost Effective 

data collection process must be efficient in both time and money. 
Quantifiable 

numbers are produced denoting the Tech Prep consortium's program development and 
progress. 

numbers convince audiences including federal, state, and local legislators to continue to 
Invest in the Tech Prep educational reform initiative. 

Transferable 

local evaluation plan should complement and augment existing federal and state 
requirements and keep additional evaluation activities to a minimum. 

program comparisons should be made with the understanding and sensitivity that not all 
consortia are Initially equal and at the same stage of development. 

provide for transferability among statewide consortia so that a consortium may draw upon 
the expertise of those who are having success In certain areas. 

4 
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Formative and Summative 

on-going evaluation (formative). 

conducted at the conclusion of a program to determine the worth of the program's 
accomplishments (summative). 



Evaluation Components 

Program accountability requires institutions to collect data on outcome indicators. Outcome 
indicators are objectively measurable statements that determine the quality, effectiveness, and goal 
attainment of a given program focus component. Outcome Indicators include, but not limited to: course 
enrollments, program completers, job placement, number of articulated courses, number of students enrolled 
in the classes, increased SAT scores, decreased dropout rates, participation of guidance personnel, and 
inservice opportunities for staff. 

Examples 

Tech Prep program completion rate 
Vocational guidance program services offered 
Tech Prep program recruitment efforts 

Program focus components are areas or components of an educational program that are to be 
evaluated. Dependent upon the nature of the program, they vary from evaluation to another. Focus 
components include: administration and organization, articulation and collaboration, student program 
planning, curriculum, marketing, program effects, ptaff development, and evaluation, it is important to note 
that most Tech Prep programs have not been in place long enough to provide completion and placement 
data for graduates, so most outcomes indicators are based on the number of articulated classes and 
students enrolled in these classes. 

Depending upon the stage of the program, the areas to evaluate may vary. The following examples 
of program components may be beneficial to consider when evaluating your Tech Prep efforts (Ruhland & 
Custer, 1994; Hammons, 1992). The most important area, for most teachers is curriculum development and 
program of study as it impacts the progress you and your program are making. Possible questions to ask 
when evaluating each component are provided. 

Administration and Organization 

Has a timeline been developed to infuse the Tech Prep program into the existing educational 
program? 

Have committees been formed with representation from business, industry, local educational 
agencies to meet and advise the consortium coordinator on activities developed? 

Who makes up the Tech Prep program staff, and what are their roles? 

Articulation and Collaboration 

Have written articulation agreements been developed between secondary and postsecondary 
Institution? 
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What is the method of articulation? 

Do articulation agreements reflect a 2 + 2, 4 + 2, or 2 + 2 + 2 program of studies? 

If articulation agreements are in place, are plans outlined to review the agreements on a regular 
basis? 

As a result of your Tech Prep efforts, what specifically are your consortium schools doing this year, 
versus last year? 

Student Program Planning and Implementation 

Has a four-year plan of study focusing on a career cluster been developed for students? 

Do counselors and teachers use program of studies to develop a students' education plans? 

What student assessment measures (GPA, test scores, program completion) will be utilized to 
document student achievement? 

What methods are planned for student recruitment and outreach? 

Curriculum Development 

What are the career clusters, if any, your consortium has developed? 

How have applied academics and workplace readiness skills been integrated into the curriculum? 

How has the curriculum changed in the occupational or technical programs at the secondary and 
postsecondary levels to reflect a2 + 2. 4 + 2, or 2 + 2 + 2 program of studies? 

Marketing Efforts 

Who is the target audience for your Tech Prep consortia? 

What promotional materials were developed to promote/market Tech Prep, and how effective were 
they? 

Do the promotional/marketing materials reflect the 2 + 2 concepts, articulation, sequence of 
courses within the vocational areas? 

What marketing efforts have worked to inform teachers, counselors, students, parents, and 
businesses about Tech Prep? 

Program Effects 

What student assessment measures will be utilized to document student achievement as a result 
of the newly Implemented curricula? 
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How has the curriculum changed in the occupational or.technical programs at the secondary and 
postsecondary levels to reflect a2 + 2, 4 + 2, or 2 + 2 + 2 program of studies? 

How many students completed "dual-credit" courses to transfer to community/technical colleges? 

What Is being done to identify test scores for students in a Tech Prep program? 

Staff Development 

Is there a plan which outlines and provides staff development activities for all Tech Prep facilitators, 
including administrators, counselors, academic and vocational faculty? 

What have been your successful accomplishments to date in implementing staff development 
activities as they relate to Tech Prep? 

What steps have been taken to involve faculty in making school changes aimed at Tech Prep? 
Evaluation Strategies 

What methods are planned for local evaluation of your Tech Prep efforts? 

What types of information would be essential for continued success with your Tech Prep efforts? 

What barriers, if any, have you encountered In Implementing the Tech Prep concept? 

Examples 

Program Component: Program Effects 
Indicator: Tech Prep program completion rate 

Outcome: Ail Tech Prep students will complete a high school program component 

Program Component: Student Program Planning and Implementation 
Indicator: Vocational guidance program services offered 

Outcome: All Tech Prep students have a four-year plan of study focusing on a career cluster 

Program Component: Marketing Efforts 

Indicator: Tech Prep program recruitment activities 

Outcome: All stakeholders understand the benefits of Tech Prep 

There are several other questions that need to be asked, based upon the stage of the Tech Prep 
program. These questions may include: 

t. Who wants to know the information? 

2. What are they interested in? What, if anything, needs to be different? 

3. Whom are we evaluating (target population)? 

4. What specific outcomes or changes, if any, are we seeking? 

5. Who should do or assist with the evaluation? 
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National Evaluation of the Tech Prep Education Program 

A national evaluation of Tech Prep education programs Is being conducted by the Mathematlca 
Policy Research, inc., Northwest Regional Educational Laboratory in Princeton, NJ. This contract was 
awarded in October 1992, by the U.S. Department of Education, Office of Policy and Planning to conduct 
the national evaluation. 



Description of the Evaluation 

The Evaluation of the Tech Prep Education Program has two primary objectives. First, it will fully 
describe the Tech Prep programs funded under the Perkins Act - documenting the number of programs, 
their characteristics, the institutions Involved, the populations they serve, and their planning and 
implementation activities. Second, the evaluation will Identify effective practices. It will document in detail 
the approaches taken by "exemplary Tech Prep programs and developing programs which show promise 
to provide guidance to other program consortia in their efforts to improve vocational education. 

The evaluation will also measure the progress of Tech Prep students in high school and in the 
postsecondary stage of the program, and compare their progress with that of similar students not involved 
in Tech Prep; this comparison will provide a context in which to analyze the achievement and outcomes of 
Tech Prep students. 

The five-year evaluation will have three major data collection components: 

• A survey of state-level Tech Prep coordinators to document the state's role In funding and guiding 
the development of Tech Prep Programs conducted twice, in the fall 1993 and 1996. 

• A survey of local Tech Prep programs, conducted annually for four years beginning in the fall of 
1993, to document their characteristics and development, 

• In-depth case studies of selected local programs, conducted four times beginning in the 1993-94 
school year, to identify and document in detail how successful programs have been planned, 
designed, and implemented. 

The in-depth study will be based on intensive on-site examination of ten local programs. These 
programs are from the states of: Oregon, Illinois, Massachusetts, Arizona, Florida, Connecticut, California, 
West Virginia, Ohio, and Alabama. The study will include site visits and interviews with program 
administrators and other staff: a longitudinal survey of a sample of two cohorts of Tech Prep students 
selected in 11th grade and followed throughout the duration of the study; and the collection of school 
records data on the academic experiences of these students at the secondary and postsecondary levels. 
Outcomes for the sample of Tech Prep students credits earned, grades and test scores, high school 
completion, postsecondary matriculation and progress, and employment and earnings-willbe compared with 
corresponding measuresfor a national sample of studentsdrawnfrom the National Educational Longitudinal 
Surveys (NELS). 
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Summary 



It is important to remember that the evaluation component is a part of the Tech Prep operational 
plan. It is also important to recognize that most educational administrators have a tendency to undervalue 
evaluation and to deprive themselves and their programs of its benefits. 

Clearly there is a variety of models to better understand what constitutes an effective program. 
There is widespread-agreement on the ideal evaluation components for Tech Prep, and this variation 
provides for designing effective programs that suit the needs of different schools and different student 
populations. 

Thus our evaluation efforts will need to continue to address a variety of issues. These issues relate 
more to the outcomes indicatorsthat need to be reported to specifically addressthe Tech Prep efforts and 
include: 

How many secondary Tech Prep students enrolled in a Tech Prep course of study at a postsecondary 
institution? 

How many secondary Tech Prep students received advanced placement or dual credit? 

How many secondary Tech Prep students completed a vocational sequential course of study? 

How many postsecondary Tech Prep completers are employed in their focused career choice? 

How many postsecondary Te^n Prep completers are employed in a related field? 

How many postsecondary Tech Prep completers continued their education ct a four year institution? 
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